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A writing assessment study was conducted in the Fort 
Worth Independent School District (Texas) in the spring of 1986 as 
part of a larger program evaluation study of the Writing to Read 
(WTR) Program. WTR is a computer-based instructional system designed 
to develop the writing and reading skills of kindergarten and 
first-grade students. Two types of comparisons were conducted to 
address the effectiveness of WTR: (1) writing skills of WTR 
participants with those of students of the same grade level in 
traditional classrooms; and (2) the writing skills of WTR 
participants with those of students whose teachers had been trained 
in Writing Process (WP) instruction. A total of 215 kindergarten and 
270 first-grade writing samples were collected. The samples were 
scored by kindergarten and first-grade teachers. The results 
indicated no differences between the writing samples of first-grade 
students in WTR and traditional classes, but first-graders in WP 
classes scored significantly higher than those in the other two 
groups. When the scores of kindergarten students in WTR, WP, and 
traditional classrooms were compared, the WTR students scored 
highest, WP students next highest, and traditional students scored 
the lowest. However, the sample selection for this comparison carried 
some gross limitations. The comparison between writing samples of WTR 
and traditional kindergartners , which did not have sample selection 
constraints, indicated no significant differences between the writing 
scores of these two groups. Appendices include: (1) instructions for 
collecting writing samples; (2) scoring criteria; and (3) examples of 
writing samples. (Author/JAZ) 
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Using Young Children's Writing Sanples In 
Program Evaluation 

Nancy K. Naron and Norbert Elliot 

Abstract 



This paper describes a writing assessment study that was conducted In the 
spring of 1986 as part of a larger program evaluation study of the Writing to 
Read (WTR) Program. WTR Is a computer - based Instructional system designed to 
develop the writing and reading skills of kindergarten and first-grade students. 
It was Implemented In 16 schools In Fort Worth, Texas, In the fall of 1985. 

Two types of comparisons were conducted to address the effectiveness of WTR 
In the area of writing skills. The first compared the writing skills of WTR 
participants with those of students of the same grade level In traditional 
classrooms. The second comparison was between the writing skills of WTR 
participants and those of students whose teachers had been trained in Writing 
Process (WP) instruction. 

A total of 215 kindergarten and 270 first-grade writing samples were 
collected. The samples were scored by kindergarten and first-grade teachers who 
received training just prior to the scoring session. A somewhat holistic scoring 
approach was used, one which focused on the content and quality of the writing 
rather than the surface features, but allowed the readers time necessary to 
decipher the phonetic spelling and other Irregularities Inherent in young 
children's writing. 

The results indicated no differences between the writing samples of first- 
grade? students in WTR and traditional classes, but first-graders in WP classes 
scored significantly higher than those in the other two groups. When the scores 
of kindergarten students In WTR, WP, and traditional classrooms were compared, 
the WTR students scored highest, WP students next highest, and traditional 
students scored the lowest. However, the sample selection for this comparison 
carried some gross limitations. The comparison between writing samples of WTR 
and traditional kindergartners , which did not have sample selection constraints, 
indicated no significant differences between the writing scores of these two 
groups . 

The methods and results of this study are compared to those of similar 
studies. In addition, the paper discusses aspects to be considered when using 
young children's writing samples In program evaluation. 
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Using Young Children's Writing Samples in 
Program Evaluation 

Nancy K. Naron and Norbert Elliot 



INTRODUCTION 

In the fall of 1985, the Fort Worth Independent School District (FWISD) 
Implemented the use of the Writing to Read (WTR) Program in 16 elementary 
schools. This program, which Is marketed by the -International Business Machines 
Corporation (IBM), is a computer-based Instructional system designed to develop 
the writing and reading skills of kindergarten and first-grade students. One of 
the main goals of WTR is to help the students develop their ability to express 
their ideas on paper. The objective of this study was to determine the 
effectiveness of the WTR Program in the development of students' writing skills, 
in comparison to students of similar socioeconomic status in other types of 
classrooms . 

Previous evaluations of WTR (Guttinger, 1986; Murphy and Appel , 1984) have 
all compared the effects of WTR with those of traditional instruction, and have 
found differences between the writing skills of the two groups. As pointed out 
by Norton (19U4), there Is always a question about what a new educational program 
should be compared with when It is Implemented. It seems inappropriate to 
compare It to the complete absence of a program, and yet, writing has not 
traditionally been emphasized In the early elementary years. Norton admitted 
that "...some doubt exists concerning what the Writing to Read program was 
actually being compared with" (p. 23), and this doubt applies to other evaluations 
of the WTR program as well. 

Therefore, a decision was made in the present study to compare the effects 
of WTR not only to those of traditional classes, but also to those of classes 
known to be teaching writing in a strong and consistent fashion. FWISD is 
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fortunate to be one of the sites sponsored by the National Writing Project. As 
part of this project, teachers receive a minimum of 30 hours of training in 
Writing Process (HP, instruction. Students of teachers trained in WP were 
selected to be a second comparison group. 

DESCRIPTION OP THE WRITING TO READ PROGRAM 

Writing to Read (WTR) is a computer-based instructional systea designed to 
develop the writing and reading skills of kindergarten and first-grade students. 
The objectives of the program are to help the students develop their ability to 
express their ideas on paper and to read what they have written. 

The program is provided within the context of a planned learning center, 
which is staffed by a full-tiae aide. The students spend from 30 to 60 minutes 
Per day. accompanied by their teacher, using the equipment and materials in the 
WTR center. This center, or lab. is organized into at least five learning 
stations according to guidelines outlined by the WTR developers. 

One of the main learning stations in the WTR lab is the Computer Station. 
The aide directs the students- activities at this station, students proceed 
through a series of ten instructional cycles which teach some , asic vocabulary 
words using a phonemic spelling system. i„ addition, practice activities are 
available on the computer to reinforce the skills learned in the instructional 
cycles. A great deal of repetition is a central part of the computer activities. 

A second learning station is the Work Journal Station. The Work Journals 
are designed to provide additional support and practice in the learning of the 
-aterial in the ten instructional cycles. Some of the pages are designed to be 
completed in conjunction with an audio tape. Tne main activity is to practice 
writing the cycle words in a number of formats. 

A third work station is the Writing/Typine Station t„ 

station. In one area of this 

station, students write stories hv h^H t„ *u 

write stories by hand, m the other area of this station, 
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students type their stories on coaputer using a word processing program. 

In a fourth work station, called the Listening Library Station, the students 
have the opportunity to Batch speech with written language. They listen to 
stories recorded at a slow pace tape while following the written text In the 
corresponding book. 

The fifth required station la the Make Words Station. Various activities 
are arranged to have the children practice Hatching letters and sounds using 
aanlpulatlve and appropriate alphabet naterlals. 

DESCRIPTION OF WRITING PROCESS INSTRUCTION 

Writing Process Is a system of writing Instruction which addresses the 
process by which students generate content. Formerly, writing was viewed as 
product-oriented. That Is, Instruction In writing typically took place after the 
writing was complete. A great deal of research that has been generated over the 
past two decades (Emig, 1977; Graves, 1981), however, has shown that writing 
Instruction conducted In that traditional fashion does not laprove student 
writing. 

The research has Indicated that for student writing to laprove, no matter 
the grade level, Instruction aust occur throughout the series of discrete stages 
In which writing develops: the initial drafting of a piece of discourse, peer 
response to first draft, revision, redrafting, editing, conferencing, the final 
draft, publishing, and evaluation. These activities can generally be categorized 
as prewriting, writing, sharing, revising, editing, and evaluating. The process 
is highly individualistic and recursive in nature. With the very young writer, 
less emphasis is placed on the revising, editing, and evaluating stages than on 
the other stages. 

The student/writer depends heavily on peer response to shape aeaning and to 
address correctness. The teacher's role is that of collaborator/editor. In this 
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way. writing process is student centered, rather than teacher centered, and a 
.rent deal of interaction occurs a.ong the students and hetween student and 
teacher throughout the writing process. As a result, the children are 
Participating in fundamental learning strategies which provide reinforcement, 
feedback, connection, and personalization (Emig. 19 7 7 ). 

writing process instruction i, the .odel heing utilized and constantly 
expanded and Improved b y the National Writing Project ,NWP, . T he flrst ^ aUa 
was at Berkeley, known as the Bay Area writing Project. There are now 146 NWP 
site, nationwide using this model. pwi S D l5 one of thMe slt „ mj ^ ^ ^ 
located in a public school district. 

A fundamental tenet of the National Writing Project is that teachers of 
writing .„st themselves write, therefore, teachers of Writing a s Process know the 
sort of problems that .11 writer, face in that they themselves are practitioners 
of their craft. In eddition. teacher, are encouraged to «k. presentations to 
other teacher, and to meet regularly i„ order to di,c„,s .„ d r6flne thelr 
interests is the teaching of writing. 

COMPARISON OF WTR AND WP 

The phiiosophy hehind WP is similar to that of WTS in that young children 
are allowed to spell phonetically, with a concern toward the „uality of the 
content of the writing over the editorial aspects (e.g.. spelling, punctuation,. 
There are major differences hetween the programs, however. WP encourages a great 
deal of social Interaction among the children end hetween teacher and student to 
help the children progress through the developmental stage, of writing. In WTR 
the social interaction aspect is minimal compered to the reliance on the computer 
and other materiel,, on the other hand. WTR includes component, designed to 
teach reading , k ill 5 . while WP focuses on the development of writing skills In 
hoth programs, students „. strongly encouraged to sound out letters when 
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attempting to write new words. 

DEVELOPMENT OP WRITING ASSESSMENT TECHNIQUES 

Early efforts toward developing empirically sound methods of the assessment 
of writing ability were generally either abandoned or considered much too 
unreliable or Invalid to yield much Information. However, In 1966, Godshalk and 
his colleagues attempted to correlate multiple choice testing of writing skills 
with direct assessment of writing. Ironically, it was in their efforts to 
validate multiple choice testing that they discovered the method of scoring that 
today is known as holistic, or impressionistic, scoring. The form of holistic 
scoring as it is presently used in massive assessment programs such as the New 
Jersey Basic Skills Placement Test and in individual program assessments 
throughout the nation was refined and disseminated by various researchers (Odell 
and Cooper, 1977; Myers, 1981; White, 1985). 

As it is presently defined, holistic scoring is conducted through a series 
of steps: selecting sample papers which exhibit a range — commonly six points—of 
writing, reproducing the sample papers, and training readers to come to consensus 
on the values of these papers. Once consensus is achieved, each of the remaining 
writing samples is scored by two readers. Papers receiving scores exceeding the 
pre-established range of agreement are termed discrepant and resolved by a third 
reader. 

CONCERNS IN SCORING YOUNG CHILDREN'S WRITING 

The evaluation of young children's writing poses some rather unique 
problems. Since phonetic spelling is characteristic of kindergurtea and first- 
grade students, for instance, a truly holistic reading is impossible . The 
writing must be "translated" by the reader. Moreover, readers must strive even 
harder to go beyond the surface features of correctness and understand that young 

. e 
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writers commonly employ irregularities in penmanship, grammar, and punctuation. 
Designers of the assessment must train readers to focus on the organization of 
ideas, for example, or extended detail achieved by the writer's fluency. Readers 
must accordingly come to recognize such values and score them appropriately. 
Hence, totally impressionistic scoring is shifted towards a more focused scoring 
so that the young writers may be rewarded for experimentation in their newly 
acquired technology. 

Too, the design of the topic or prompt for the writing assessment is a 
unique problem. Since formal evaluation of this population is such a new 
concept, there are presently no studies which deal with young children's 
abilities to handle subtleties of audience and rhetorical B odes of composition. 
Therefore, researchers are safest to design prompts which evoke the narrative 
patterns common to children's speech: those which evoke concrete, narrative 
experiences which are believed to be common to the cognitive capabilities of 
beginning writers. 

OTHER EVALUATIONS OP WRITING TO READ 

The first formal e ra pirical evaluation of WTR was conducted by the 
Educational Testing Service (BTS) from 1982 to 1984 (Murphy and Appel , 1984)> and 
financed by IBM. The writing assessment study that was part of their larger 
evaluation compared some 6,000 writing samples from WTR and non-WTR children and 
found that the WTR students "clearly surpassed comparison students in writing 
performance" (9.4). A i th0ugh the ETS study followed the holistic scoring 
procedures and concerns for scoring young children's writing outlined above, 
there has bee n some criticism of several other aspects of the ETS study 
(Hathaway, 1985) . 

other evaluations of WTR that have included writing assessment, (Guttinger, 
I***; Kara.. l 9 ae, have not utilised the ri 8 orous procedures of the colleotion 
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and scoring of writing samples that were used in the ETS study and in the present 
study. The present study differed from the ETS study in one important aspect 
that is described in detail in the section which reports the procedures used in 
the collection of writing samples. 

IMPLEMENTATION OF WTR IN FORT WORTH SCHOOLS (FWISD) 

Beginning in the fall of 1985, the WTR program was implemented for the first 
time in 16 elementary schools. In nine of these schools, the program was 
implemented only with kindergar tners . In* five of the schools, both 
kindergartners and first-graders were Included in the program. At one school, 
multi-age classrooms (four- and five year-olds together and first- and 
second-graders together) were included, while at another school, the program 
included only first- and second-graders. 

Description of the WTR Schools . Of the nine schools using the program with 
kindergartners only, all but one were minority schools with at least half the 
children on the free and reduced meal plan. None of these schools had more than 
four percent of its students who were limited English proficient. 

Of the remaining seven schools that used the program with first-graders as 
well as with other grades, the profile of the schools was much more varied. The 
percent of students on the free and reduced meal program ranged from 21 to 81 
percent, and the percent of students classified as limited English proficient 
ranged from zero to 76 percent. 

Cost of the Program. The total cost of the WTR program for the first yeax* of 
implementation in 16 schools was $440,50*°, for a cost per student of $236. The 
cost covered the following items: salaries for a full-time aide in each of the 16 
schools, 8 IBM PC jr3. per school, WTR program software, teacher manuals, student 
work journals, tape recorders, and other materials and equipment required as part 
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of the prograa. 

The cost of the program for subsequent years is estiaated to be about half 
the cost of the first year, to cover the recurring costs of the aides' salaries, 
the student work journals, and aaintenance, repair, and replaceaent of equipment. 

IMPLEMENTATION OF WRITING PROCESS IN FWISD 

Unfortunately, a very Halted nuaber of teachers at the kindergarten and 
first-grade level had been trained in WP in FWISD at the tiae of this study, m 
fact, only one kindergarten and one first-grade teacher wers identified. i„ 
order to gather aore data for this coaparison. the evaluation teaa decided to 
include two first-grade classrooas in Arlington I.S.D. whose teachers had 
previously received WP training. 

Description of the WP Classes. The kindergarten WP classrooa was in a school ln 
which acre than half the students were on the free/reduced aeal plan and over 30 
percent of the students were Halted English proficient. The first-grade WP 
class in FWISD was in a school of the saae profile as that of the kindergarten 
class. The two first-grade VP classes in Arlington I.S.D. were in schools with 
relatively few children who were on the free/reduced aeal plan or who were 
Halted English proficient. 

Cost^ The only expense incurved in iapleaenting WP in a classrooa was the 
training of the teacher. A 30-hour training session was the ainiaua required, 
estiaated to cost approxiaately $32 or less per teacher. 

THE FWISD WRITING ASSESSMENT STUDY 

Design of the Study . Because of the liaitation in the nuaber of available early 
childhood WP classrooas. It was necessary to aat.h WTR and traditional classrooas 
to those WP classrooas. Conducting a study with aatched groups has a nuaber of 
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drawbacks. For one thing, due to the lack of pre-treataent measures related to 

thw program objectives, the selection of characteristics on which to Batch Is 

necessarily questionable. Even in the aost ideal istically perfect natch between 

groups on & characteristic such as socioeconoaic status, the researcher cannot be 

certain that the groups were Batched on a characteristic that equated the groups 

with respect to the outcoae aeasure. And in the present study, the Batches on 

the selected characteristics were far from perfect. When coiaparisons are between 

groups that were different or unequal before the treatment, and when post-treat- 

aent differences in the dependent variable are found, one cannot determine 

whether these differences are a result of the treatment or due tc the prior 

inequalities. As Cronbach (1963) points out: 

Any failure to equate the classes taking the coapeting 
courses will jeopardize the interpretation of an ex- 
periment and such failures are alnost inevitable, (p. 49) 

Despite the probleas caused by the use of non-randoaly selected comparison 
groups, the evaluation team felt that the collection of any inforaation relative 
to this coaparison, no natter how Halted, was preferable to no such comparison. 

In addition to the one kindergarten and three first-grade HP classrooas each 
being Batched to a WTR and a traditional classrooa, additional WTR and 
traditional classrooas were selected to represent the distribution of WTR 
classrooms across the district. A total of 24 classrooas were selected to 
include in the writing assessaent study: 10 WTR, 10 traditional, and 4 WP 
classrooas. Table i presents the design of the study. 

The schools were matched as closely as possible on two variables: (1) socio- 
econoaic status, as deterained by the percent of students on the free and reduced 
aeal prograa, and (2) the percent of students classified as Halted English 
proficient (LEP). Once the school sites were identified, individual classrooms 
were selected according to the following guidelines: (1) no ESL or bilingual 
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Collection of Writing S apples . Writing samples were collected from each of 
the 24 classrooms Included In the study, with a total of 215 kindergarten and 270 
first-grade samples collected. All samples were collected during the morning of 
Thursday, May 1, 1986 9 with the exception of two classrooms whose samples were 
collected on the morning of Hay 2, 1986. 

The procedure followed for the collection of writing samples was very 
similar to that used In the ETS evaluation of WTR (Murphy and Appel, 1984) with 
one Important difference. Unlike the ETS study in which the classroom teachers 
administered the instructions and collected the samples , outside evaluators In 
this study conducted the actual administration of the instructions and collection 
of writing samples. This procedure had several advantages over the ETS approach. 
All the evaluav participated In a 45-minute training session to assure some 
uniformity in the collection procedure. In addition, the presence of outsiders 
offered some assurance that the teachers In fact did not assist their students In 
the writing task. (In one classroom, a teacher was observed assisting two 
students, whose samples were not Included in the study.) 

A total of 12 evaluators participated in the collection of writing samples. 
Each evaluator collected the writing samples In two classrooms. To avoid any 
systematic differences In the administration of the Instructions, all but two of 
the evaluators were assigned to a classroom in two different treatment groups. 
For logistical reasons, two of the evaluators collected samples from classrooms 
in the same treatment groups, but both had had extensive experience with 
elementary students. 

The instructions used to elicit the writing samples (see Appendix A) were 
developed by one of the co-authors who had worked at ETS at the time of the 
original evaluation of WTR (Murphy and Appel, 1984). The Instructions were very 
similar to those used in the ETS study. If for any reason the instructions used 
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to obtain the writing samples were biased against any of the children because of 
their socioeconomic status or their language background, the effects would have 
been equally distributed across the treatment groups as a result of the matching 
design. 

The instructions were followed exactly as listed in Appendix A by all evalu- 
ators with one exception: the time allowed for completing the writing task. In 
10 of the classrooms, a few children took longer than the 30-minute period: 
children in four classes took up to 35 minutes, children in another five class- 
rooms took up to 47 minutes, and one child took an hour and 15 minutes to 
complete the task. Of these 10 classrooms, five were WTR classes, two were WP 
classes, and three were traditional classes. Evaluators stated that they rushed 
a total of 17 students to finish their stories, an average of 1.8 from nine 
different classrooms. The approximate average amount of time that students from 
different treatment groups took to complete their stories is listed in-Table 3. 

The evaluators were asked to describe the students- responses to the writing 
assignment. Overall, the response was as expected, with WTR and WP students 
excited and comfortable with the task, while students in traditional classes were 
somewhat less excited and aore confused by the task. There were some notable 
exceptions to this trend, however. In two of the WTR kindergarten classes, the 
students acted as if they did not know what to do, and in three of the 
traditional first-grade classes, the students were quite excited and obviously 
experienced in creative writing tasks. In addition, the teacher-s response in 
one traditional kindergarten class was fascinating: she said that she had never 
asked her students to write stories and was shocked to see that they could. When 
the evaluator asked the students to read their stories to her, the teacher 
thanked the evaluator because the teacher felt she herself would not have been 
able to decode their phonetic spelling. 
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The role of the classroom teacher during the writing assignment varied 
somewhat. All the teachers were asked to remain in the classroom during this 
time and to write the lead sentence on the board. Beyond that, the teachers 
chose different levels of Involvement within our guidelines: some Just did work 
at their desks, some helped pass out writing paper, some answered questions, some 
walked around the room offering encouragement to the students. There was no 
discernible trend In the teachers' behavior according to treatment group or age 
of student. Of the 24 classrooms, three had substitute teachers that day: one 
first-grade traditional and two first-grade WTR classrooms. 

The role of the evaluator was quite specific and Is outlined In Appendix A. 
In addition to these written guidelines, all the evaluators were Instructed to 
walk around the rooa during the writing assignment and offer appropriate encour- 
agement. Most of the evaluators accepted completed papers before the end of the 
30-mlnute session; two evaluators left the papers with all the students for the 
full 30 minutes. * 

Scoring of the Writing SaBples . The scoring procedure utilized In this study was 
the same as that used in the ETS evaluation of WTR (Murphy and Appel, 1984). As 
described earlier, this approach is somewhat holistic in nature, but allows for 
the decoding of individual words which are spelled phonetically. Overall, the 
scoring criteria do not include penmanship, spelling, grammar, or punctuation. 
Rather, the criteria address aspects such as the development and organization of 
ideas, and other standards reasonable to apply to the writing of young children. 

A team of four people (a writing consultant who worked at ETS at the time 
of the WTR evaluation, a FWISD writing expert, and two FWISD evaluators with 
extensive early childhood experience) reviewed the writing samples to establish 
the criteria for scoring, without knowledge of each sample's origin. The 
kindergarten samples were examined separately from the first-grade samples. The 
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review team read through all the kindergarten samples and a random group of over 
half the first-grade samples to establish the criteria. The process was the saae 
for both sets. Upon reading a sample, each reviewer would decide at which level 
that paper might fall on a scale from zero to six. As each team ae mber read more 
papers, he or she would begin to formulate a concept of what aspects of a paper 
would determine the level of that paper. All team members discussed their ideas 
throughout the process and slowly refined the criteria with team consensus. The 
final kindergarten scoring criteria are listed in Appendix B, and the final 
first-grade scoring criteria are listed in Appendix C. 

Once the scoring criteria were established, the review team selected a 
representative writing sample for each scoring level wixhin each grade level to 
serve as an illustration of that level. The example kindergarten writing samples 
are included in Appendix D. and the example first-grade writing samples are in 
Appendix E. 

Kindergarten and first-grade teachers from across the school district were 
solicited to be scorers of the writing samples, a letter requesting teachers' 
participation was sent to every kindergarten and first-grade teacher of the 45 
non-Writing to Read schools. (No teacher from a school included in the study was 
to be a scorer. Due to timing limitations, the evaluation team did not know 
which WTR schools were to be included in the study at the time the letter of 
request for scorers was sent.) A total of 111 teachers responded to the letter. 
Of these. 26 were from schools included in the study and could not be scorers. 
Of the remaining 85. 50 were selected to participate. They were selected so as 
t* - resent as many different schools as possible and then randomly within this 
(W'eHne. Of these 50. 38 actually participated in the scoring procedure, 
rfceresAfttlng 25 different schools. Of the 38 scorers. 21 were kindergarten 
tMkfate ' and 17 We " "^-grade teachers. Because of the larger number of 



19 



15 



first-grade samples, six of the kindergarten teachers scored first-grade samples 
while the rest scored samples of the grade level they taught. 

None of the scorers knew the exact purpose of the activity, other than as a 
study of the development of writing skills of young children. After an Introduc- 
tion, the scorers of kindergarten samples were separated from the first-grade 
scorers. Each group was then trained on the scoring procedure. Each scorer was 
given a copy of the set of samples selected to represent the seven different 
levels f-~ grade level (see Appendices D and. E). The samples were coded so 

that the uconus were unknown, and placed In the following order In each packet: 
the "6" sample, the "1" sample, the "5" sample, the "2" sample, the M 4" sample, 
the W 3 W sample, and the "0" sample. The scorers were Instructed to read each 
sample in turn and to place them In piles of "high" and "low" quality. Following 
this, the scorers were asked to rate each paper with a score from zero to six, in 
a comparative fashion. 

When the group had completed this activity, a hand count was taken to 
determine how the scorers rated each sample. Appendix P presents this tally of 
the training samples for the kindergarten scorers, and Appendix 6 presents this 
tally for the first-grade scorers. In only one case did any acorer deviate more 
than two points in either direction from the intended score. The level of 
consensus for the Intended scores ranged from 62 to 94 percent for the kinder- 
garten samples, and from 58 to 100 percent on the first-grade samples. A group 
discussion of the merits and limitations of each paper followed this tally. 

Upon completion of this exercise, each scorer was given a folder with 
writing samples — each of the 15 kindergarten scorers received a folder with 14 
or 15 random writing samples, and each of the 23 first-grade scorers received a 
folder with 11 or 12 random samples. Each sample was coded In such a way that 
the treatment group and school were unknown to the scorers. The scorers were 
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instructed to use the guidelines outlined for that grade level for scoring (see 
Appendices B and C) as well as the sample papers. They were told to ignore 
pen B anshl P , grammar, punctuation, and spelling, and to not compare papers within 
the folder but rather to compare each to the guidelines and samples. For each of 
the groups, two BeB bers of the original review tea. were available to answer 
questions and help scorers to decode words that were spelled phonetically. 

Each paper was read and scored Independently by two readers. Whenever the 
two scores differed by three or B ore points, (e.g., 6-3, 5-2, 4-1), the paper was 
coded as "discrepant" and read by a third reader. Of the 215 kindergarten 
samples, 10 (or less than 5 percent) were scored dlscrepantly , and of the 270 
first-grade papers, 20 (or 7 percent) were scored dlscrepantly. In both cases, 
this Is rather high Inter-rater reliability. 

The final score for each s aB ple was the SUB of the scores given by the two 
readers. In the case of a discrepant paper that was read by a third reader, the 
two elect scores were used to deter B lne the sable's final score. Thus, the 
range of scores for each sample was from zero to 12. 

Results of the Writing ^ess.ent. The results of the writing assess B ent were 
examined In three ways at each grade level: (i) comparing scores of students i„ 
matched WTR, WP. and traditional classrooms, ( 2 ) sparing scores of students In 
WTR and traditional classrooms that were not matched to Writing Process classes, 
and (3) comparing scores of all WTR and traditional students in the study. 

The mean scores on the writing assessment for the students in the matched 
comparison of WTR, WP, and traditional classes are presented 1„ Table 4. 
Cautions concerning this comparison BU st be emphasized. The matching procedure 
alone is problematic, as discussed ln the section on the design of the study. In 
addition to the statistical and design problems built into this procedure, the 
range of demographic characteristics of the WTR schools was so limiting that some 
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of the Batches were questionable (see Table 2). A further concern Is that even 
though the level of analysis Is at the level of the student, In fact only one 
kindergarten class and three first-grade classes were Included in each of the 
appropriate cells. Given the important effect that the teacher has on outcome 
measures such as that used here, this is a considerable limitation. In addition, 
two of the first-grade WP classes were in another district, and the teacher in 
the third WP first-grade class was out on maternity leave for eight weeks during 
the school year. 

Because of all the limitations on these data, no tests of significance were 
conducted. At the kindergarten level, the trend in the data indicated that WTR 
students scored the highest, WP students next highest, and the students from the 
traditional class scored the lowest cn the writing assessment. Not much weight 
should be placed on these findings, however, because the comparison was based on 
only one classroom for each treatment, and the demographic match between the WTR 
class and the WP and traditional classes was poor. At the first-grade level, the 
students from the WTR and traditional classrooms scored about the same on the 
writing assessment while the WP students scored considerably higher. 

Tables 5 and 6 present these data in a different format: the range and 
frequency of scores received by each group. In Table 5, which presents the 
kindergarten scores for the matched groups, the most obvious feature is the small 
range of scores (from zero to two) in the traditional classroom. The main 
difference in scores between the WTR and WP classrooms is the much larger number 
of samples scored "zero" in the WP than in the WTR classroom. It must be 
reiterated that while the kindergarten WP and traditional classrooms were from 
the same school and therefore well matched on demographics, the match between the 
kindergarten WTR and WP classroom was probably the most disparate (see Table 2), 
with the WTR class from a much higher socioeconomic status school than the WP and 
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traditional classes. Therefore, it was difficult tn a 

, « was difficult to draw any generalizations 

froa these limited data. 

Table ., which presents the (taHn(l scores f(jr Mtchcd ^ js 
based on .ore data than „ ^ klndergartcn level ,„ ^ 
of score, f „ r the three .roup,, one .... tfc . t the OTR and tradltloMl 
distribution, were eesentially tbe saae. with tbe .ajority of tbe ecore, falling 
in tbe low to a.ddle range (fr0 . 2 t0 8) . Tne „ ulk >f ^ ^ ^ 
the other hand. ... definitely In tbe upper range, of ecore, (fro. 7 to .„ One 
nust reae.ber. however, that two of tbe three WP clessrooa, were locate d in . 
neighboring district and .ay no t be operable to our schools. Ih e,e echool, 
,erved high eocloecono.lc etudent,. and they were .etched to the PW.SD ,chool, 
that eerved tbe hlgheet ,ocloeco„o.lc students available within the treataent 
group,, Nevertheless, a, with tbe Kindergarten watched data, it wa, difficult to 
-eke an, definitive stateaents about the flret-grade .etched ccparisons. 

' The co.parisons between the scores of ,tude„t, in „Tg and in traditional 
cl.ssroo.s not .etched to WP carried fewer ligation, and were .ore cleercut in 
their reeult,. Ta ole 7 preeent, tbe .een eccre, for tbe group, i„ the8e 
coperlsous. T e,t, of ,ig„ificance that were epplied to eech ccparlson indi- 
cated no significant difference between the writing saaples of students In ra 

end thee In traditional claesroo.e at hoth the Kindergarten and flret-grade 
level. 

Tables e a „d 9 prc ,e„t tbe,e data in teres of the range end frequency of 
ecore, for each group. « th e Mndergerten level (Ta hle 8,, the .cat interring 

36 percent for the WTH etudent, ccpared to 45 percent for tbe non-bra student, 
0. the other hand. 18 percent of the WTR students received a score of four or 
above ccpared to 25 percent of the non-WTR student,. 
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In the sose comparison at the first-grade level, as depicted in Table 9, 
very few differences were obvious. Twenty percent of the WTR students received a 
score of zero on the writing assessment compared to five percent of the non-WTR 
students. Other than that, the distributions were very similar. 

In the third overall comparison, the scores for all the WTR students in the 
study were compared to the scores for all the students from traditional class- 
rooms in the study. In other words, this comparison combined the students from 
the first two overall comparisons, but eliminated the students from the WP 
classrooms in the comparison. Table 10 presents the means and standard devia- 
tions for this comparison. 

A test of significance at the kindergarten level indicated a significant 
difference between the scores of WTR students and those in traditional class- 
rooms. This is not surprising when one realizes that the matched WTR score was 
the highest of all and its matched traditional score was the second lowest of all 
classes. These two outlier scores changed the overall complexion of the compari- 
son considerably, but because the saoe limitations of non-random assignment 
apply, this comparison must be viewed with great caution. At the first-grade 
level, the difference between WTR and traditional students remained non-signifi- 
cant. 

As the standard deviations indicated (see Table 10), there was a great deal 
of variance among the various classrooms included in the study, particularly at 
the kindergarten level. Table 11 presents the mean scores for the individual 
kindergarten sites, and Table 12 presents the mean scores for the individual 
first-grade sites. As one can see, the variance within each group was at least 
as large as the variance between groups, particularly at the kindergarten level. 
These data reinforce the cautions with which one should view the writing assess- 
ment results. 
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Discussion of the Writing Assent Results . Overall, when reviewing the 
results of the writing assessment study, one must remember the strong Imitations 
of the study design: a small sample size and non-randomly selected comparison 
groups. However, within these constraints, the results are straightforward with 
regard to one comparison and suggest trends on the other comparisons. The one 
straightforward result which was supported throughout the analyses is that there 
was no difference between the writing samples produced by the first-grade WTR 
students and those produced by the first-grade students in the traditional 
classes. Even within the constraints of the matched group design, assuming that 
there were inequalities between the groups prior to the treatment, one would 
strongly expect that the presence of a treatment such as WTR compared to no such 
treatment would produce some effect on a direct outcome measure. No such effect 
was even suggested in the first-grade data. Based upon these results, in 
conjunction with the observations of the evaluators who collected the writing 
samples, one might conclude that traditional first-grade teachers may in fact be 
Including creative writing as part of their curriculum, at least within the 
confines of the type of first-grade classrooms included in this study. 

The data from the Writing Process classrooms at the first-grade level 
strongly suggested an effect from this type of instruction, compared to that used 
in WTR and traditional first-grade classrooms. However, the effect was strongest 
from the schools serving the highest socioeconomic status students, so that any 
generalizations were limited. Nevertheless, the data indicated that this cost- 
effective alternative to improving the writing skills of young children offers a 
great deal of promise and should be examined in greater depth. 

At the kindergarten level, the only comparisons which supported the positive 
effect of WTR on the development of writing skills were those which included the 
matched classrooms in the comparison. The WTR school i„ this match served much 
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highca* .ncl economic status students than the school aatched to it, and the 
resulting scores placed the students frou these two schools at the two ends of 
the writing assessment continuum. 

When these poorly Batched outlier classrooms were not Included in the data, 
the results indicated no significant differences between the writing skills of 
kindergartners in the WTK program and those in traditional classrooms. However, 
with the wide variance within groups as well as between groups, one might 
conclude that the individual teacher had at least as much effect on the writing 
skills of his/her kindergarten students as the presence of the WTR program. 

In the comparison of WP with WTR and traditional writing instr tion at the 
kindergarten level, again only a trend was suggested. This *ison was 

limited to only one classroom per group and suffered from the poor match between 
the WTR school and the school in which both the WP and traditional classrooms 
were located. However, the match between the WP and traditional classrooms, 
being from the same school, was as good as it could be on the population demo- 
graphics. This comparison suggested a positive effect when the teacher had been 
trained in WP, but again, more extensive investigation is necessary before any 
definitive conclusion is possible. 

It should be noted that the findings of this study differed from those of 
the ETS study of WTR (Murphy and Appel, 1984), in which differences in writing 
ability between WTR students and students in traditional classrooms were found. 
The only difference between these two studies in the implementation of the 
writing assessment study was in the data collection procedure: in the ETS study, 
the teachers themselves collected the samples, whereaa in the present study, 
outside evaluators collected the samples. The latter procedure insured the 
validity of the samples. 
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CONCLUSION 

It mist be recognized that It is somewhat difficult to plan, execute, and 
analyze the results of a well-designed writing assessment study. Nevertheless, 
the Information obtained from such an endeavor Is Invaluable to evaluators of 
programs which purport to inwove the writing abilities of the participants. We 
hope that procedures outlined In this paper will both encourage and guide 
other school district personnel to utilize direct assessment procedures for 
evaluating the development of writin-j skills in their students. 

We would like to suggest a few general guidelines in conducting such an 
endeavor. One of these is to use a team approach In designing a writing 
assessment study. In our study, combining one team member's expertise In 
teaching and assessing writing with another's expertise m the creation of a 
workable and valid research design resulted in a far more Informed approach 
toward evaluating writing than either member could have achieved individually. 

Second, those who undertake this type jf study should be aware that a well- 
planned writing assessment must be carefully designed far In advance of the 
actual scoria of the witing samples. The creation of the prompt, its 
administration, the collection of the papers, the selection of sample training 
papers, and the definition and refinement of the scoring criteria are not mere 
mechanical duties to be executed quickly, but rather are significant variables 
that can affect the validity of the results If not implemented with care. For 
example, r believe that the administration of the prompt and the collection of 
the writing samples b- outside evaluators rather than by the classroom teachers 
was an es^tial ele* int in the obtainment of writing samples that truly 
reflected the skills of the writers, m addition, when the steps of the writing 
assessment are applied to the collection of writing samples from young children, 
extra caution oust be used to address the particular idiosyncracies and needs of 
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this age writer. 

When school district personnel design, administer, and analyze their own 
direct writing assessments, rather than using some machine scored method provided 
by a "professional" testing agency, all those Involved benefit from some 
wonderful by-products. The school administrators are able to Inexpensively 
assess writing with the active participation of the district's teachers 
themselves. The teachers who conduct the scoring gain a sense of method through 
which they themselves can assess the work of their , students and Integrate it Into 
their regular instruction on an ongoing basis. The students of these teachers 
will benefit from both their teacher's Increased insight and skills and from 
their own Increased sense of what Is realistically expected and possible in 
successful writing. Such an awareness is fostered at a staple level when their 
teachers share with them sample training papers which exhibit the characteristics 
of effective writing by other students of their grade level. At a more complex 
level, the students will have Increased awareness on an ongoing basis as their 
teachers begin to tie the assessment methods to their instruction in writing. In 
this way, connections between program evaluation and program development are 
formed . 

Indeed, the use of direct assessment methods of evaluating writing can 
create a sharing and refinement of knowledge among Individuals that will 
contribute toward a more Informed and richer experience for our students. 
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Table 3 



Estimated Mean 


Number of Minutes for 


Students in Different 


Treatment Groups to 


Cor-mlcte ^heir 


Stories 








Writing to Read 


1 Writing Process 


1 Traditional 


Kindergarten 


12.2 


1 15 


I 13.4 


First-Grade 


24 


1 23.3 


I 17.4 



Table 4 

Mean Writing Assessment Scores for Students in Classrooms Matched to Writing 
Process Classrooms 





1 Writing to Read 


Writing Process 


Traditional 




1 6.67 


I 3.21 


1 0.61 


Kindergarten 


1 (N=21) 


(N=19) 


fN-181 




I 4.85 1 


7.89 


4.72 


First-Grade 


1 (N=53) 1 


(N=71) 1 


(N*64) 
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Table 1 



Number of Classrooms from Each 


Treatment Groun Selected for the 


Study 


Writing to 


Read ! Writing Process ! 


Traditional 


Kindergarten 5 


1 i 

! 1 

! i 


5 


First-Grade 5 


! 3 ! 

! i 


5 


SSaSBCB88;:aSSS8S3=BSBttBSSSS3BSS 







Table 2 



Demographic Description of the Schools included in the Writing Assessment Study 



Writing to Read 



Writing Process 



KINDERGARTEN 







FRM 9 


LEP b 


Site 


1 


: 26 


12 


Site 


4 


67 


76 


Site 


6 


85 


4 


Site 


8 


51 


0 


Site 


10: 


57 


0 



SSite 2 : 



FRM LEP 
59 21 



i l^aai 1 


Clonal 
FRM 


LEP 


(Site 3 C 


: 59 


21 


.'Site 5 


77 


53 


ISite 7 


85 


5 


iSite 9 


51 


1 


ISite 11: 


63 


2 



FIRST-GRADE 



Site 


12 


: 38 


21 


ISite 


13 d : 


57 


21 


ISite 


14 d 


57 


21 


Site 


15 


: 24 


7 


ISite 


16 : 


10 


3 


ISite 


17 


: 12 


2 


Site 


18 


21 


0 


ISite 


19 : 


3 


0 


ISite 


20 


14 


0 


Site 


21 


60 


27 










ISite 


22 


60 


14 


Site 


23: 


81 


44 










ISite 


24 : 


75 


46 



a) the percent of students in that school on the free and reduced meal program 
(FRM) p 8 

b) the percent of students in that school identified as limited English 
proficient (LEP) 



c,d) different classrooms from the same school 
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Table 5 

Distribution of Kindergarten Writing Scores for Classro oms Matched to Writing Process Classrooms 



Frequenc y of Scores by Treatment Grou^ j 



Scores 



1 > 


Writing to Read 


/ M O -« \ I 

(N=21 ) 1 


Writing Process (N=19) 


1 Traditional (N=18) 


1 { 

12 1 i 


X 






I 0 






I \ 

11 ! * 


X 






! — 




! 
1 


i I 

i i 

io : i 


X X 


X 


X x • 


10 




J 


i : 

9 ! I 














: I 

8 ? ! 


X X 


X 


X 


t 0 




; i 


i i 

i i 

71 1 


X 












1 1 
I i 


X X 






! 0 


0 




! 1 
5 II 








i 0 






i i 
i i 

4 M 


X X 






1 0 






! I 
3 I I 


X 






! 0 


0 




! I 
2 I I 


X X 






! 0 


0 


I * * ♦ * 


! 1 
1 I I 


X 










\ * ♦ ♦ 


I! 














0 ! I 


X 






1 0 


0 0 0 0 0 0 0 


| *********** 
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fable 7 



Mean Writing Assessment Scores for Students In Classrooms 
Not Matched to Writing Process Classrooms 





r Writing to Read 1 


Traditional 




! 2.25 I 


2.00 


Kindergarten 


(N=81) 1 


(N=76) 




1 4.13 ! 


4.57 


First-Grade 


(N=*40) ! 


(N-42) 
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Distribution of First 
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Table 10 



Mean Writing Assessment Scores and Standard Deviations fo r 
All Study Participants in WTR and Traditional Classes 





! ! Writing to Read ! ! 




Traditional 




I ! Mean ! S . D . ! N ! ! 


Mean 


I S . D . ! K 


Kindergarten 


I ! ! i ii 

II i i if 

i : : : 1 1 
M i i : ; 

1 3.16 1 3.15 ! 102 1 ! 


1.73 


1 1 
I 

I I 

! 2.37 I 94 


First-Grade 


! ! 1 1 II 

N : i ! i 
ii i i ii 

1 1 4.54 ! 3.02 ! 93 II 


4.66 


! 1 
! 1 
1 

1 2.82 I 106 
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Table 11 



Mean Writing Assessment Score for Each Kindergarten Site 



Writing to Kead 


I Writing Process 'l Traditional 


Site 1: 6.67 


1 

1 Site 2: 3.21 1 Site 3: 0.61 


Site 4: 1.74 




Site 5: 1.58 


Site 6: 1.61 


Site 7: 0.33 


Site 8: 3.91 


Site 9: 4.21 







Site 10: 1.67 



! Site 11: 2.06 



BaasscasssaasssBsss^sasns&assscssssssssssassasssassasssass 



Table 12 



Mean Writing Assessment Score for Each First-Grade Site 



Writing 


to Read 


1 Writing Process 


Traditional 


Site 12: 


6.29 


! Site 13: 


5.00 


! Site 14: 


5.00 


Site 15: 


3.94 


1 Site 16: 


8.85 


Site 17: 


4.53 


Site 18: 


4.39 


1 Site 19: 


9.04 


Site 20: 


4.61 


Site 21: 


5.19 




1 


Site 22: 


5.45 


Site 23: 


2.95 






Site 24: 


3.77 
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Appendix A 



Instructions foy y Collection of Writing Samples 



1. Materials ; Students should use the writing Implements that they normally 
use and the Primary Writing Tablet paper that has been provided. If they 
do not have anything to write with, supply each with a pencil. 

2. Length of Time ; After the directions have been given, allow the students 
30 minutes to complete their story. If they finish early, collect the 
sample and give those who are finished extra paper for drawing a picture. 

3. Length of Response : There is no required length. 

4< Te acher Assistance : Please do not assist the students in any way. If the 
student needs more than one sheet of paper, provide that, but do not 
staple the pages together until after the 30 minute period. 

5. Directions : After the pencils and paper have been distributed, give the 
students the following directions: 

(Say) Today I'm going to ask you to write a story all by 
yourself. I want yon to tell a story about a make believe ride in a magic 
car. You can do anything you want to on the ride. You can drive to the 
moon, or you can drive back in time to see the dinosaurs. You can drive 
to visit someone that you like or you can go to eat chocolate Ice cream 
with Big Blvd. You can pretend to go anywhere that you want. 

Now, I want you to write a story all by yourself about what you would do 
if you took a ride In a magic car. 

/our teacher is going to write the beginning of the story on the board: 

(Print out this stem on the board): 

Once I took a ride In a magic car. 

(Read this sentence aloud to the children.) 

(Say) Now you write the rest of the story yourself. Try ltl 

6. Special Circumstances : 

a) If any children ask you to repeat the directions, or say they do 
not know what to do, repeat the last sentence of the prompt: 

"I want you to write a story all by yourself about what you 
would do If you took a ride in a magic car." 

If any children ask If they can draw a picture, say: 

That's o.k., but we'd like you to write a story too if you can." 
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If any child becomes visibly upset by the task, you may go to that 
individual child and say quietly: 

"It's o.k. to draw a picture with your story if you want to." 

If any children ask for help from other students or the teacher, say: 

"Try to do it by yourself." 

Concluding the assessment ; At the end of the 30 ninute period, take 
up all papers. If the student has used more than one sheet r staple the 
sheets together. Staple the index card with the code facing down at the 
sane time. Put the staple at an angle in the top left hand corner. Thank 
the children and the teacher for their time and cooperation! 
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Appendix B 
Kindergarten Scoring Criteria 

Level 6 

The writer clearly conveys a narrative account through vivid or striking word 
choice. The writer may also indicate an awareness of the cause and effect 
relationship of events. 



Level 5 



The writer conveys a narrative account through clear word choice. 



Level 4 



The writer conveys an event or events, but these may be somewhat disjointed. 



Level 3 



The writer attempts to convey an event or events but often falls into a pattern 
of repetition or listing. 



Level 2 



The writer establishes a direct relationship with the stem, but little or nc 
detail is given. 



Level 1 



There is some attempt to write, but there may only be one intelligible word 
beyond the stem. 



Level 0 



The stem is merely copied, or the stem is copied and a picture is drawn. 
Unintelligible words may follow. 
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Appendix C 
First-Grade Scoring Criteria 
Level 6 

The writer clearly conveys a narrative account through vivid or striking word 
choice. Details are essential to the sense of a narrative. 

Level 5 

The writer conveys a narrative account through clear word choice. Details 
contribute to the sense of a narrative. 



Level 4 



The writer has a simplistic form of narrative, although details may be somewhat 
additive or repetitious 

Level 3 

The writer attempts to convey a sense of story, but the story is often repeti- 
tious in its detail. 

Level g 

Some details are added beyond the stem, but these may be very disjointed. 

Level 1 

Merely a comment or a list of words follows the stem. 

Level 0 

Material following the stem does not constitute an understandable sentence. 
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Appendix D 



Examples of Kindergarten 
Writing Samples 
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a 



ERIC 



Kindergarten Writing Sample Score of 2 



Or\ce~L\oo\ QrlJa 
in awacj\Gcar 

c arlwert ±o filestore cauomfi * « r r fl / ( 
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I 

• V 



4 



i 



I 



ERIC 
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Kindergarten Writing Sample Score of 6 




CqodAy. 1,„ hnoTiOlL 



m 
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Appendix E 



Examples of First-Grade 
Writing Samples 
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First-Grade Writing Sample Score of 2 





{ )n('jp. 








izfcidp in rm mncft' 








~<n-\r 








my f riei rids 




rrrwir\eM&< 


Tvertrr: 


















-fchpzs— — . ■ ■ -:. - 






•I 











69 



First-Grade Writing Sample Score of 3 



Page 1 of 2 



\oy.\jidi>:. 
Once: I took a ride 




m:Q..mag\c. con. a 
I waeol" tcSgBSs 
b oo se <W we- wait 



to Jib 



ml 



we wcEnt. dto_see.ny 
m.. .ma .we.ywt 





€sm see ny cla< 
we. woenf .To seer^ 
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Page 2 of 2 

sister ond we \yont)o 
see my brother cm 
we waenf ..to ..see 
B i g Bird s ..mow and 
dao and sise-c and 

hrother . and. we 
vweot to. get. same 
tccrew ond. we. 
bod ' fun. ' 

The. End. 



63 



71 



First-Grade Writing Sample Score of .4 Page 1 of 2 




(j 
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65 




86 
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Page 2 of 2 




r. . 




3E 



EES 




in 






now:' 



ERIC 
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First-Grade Writing Sample Score of 6 



Page 1 of 4 



It 



I'fe lib 




zz ... .. , 



. - „ ...... n r-r^ — 5 SS&figfrj 



L2: 
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Page 4 of 4 




ERIC 
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Appendix F 

Tally of Scores on Kindergarten Training Sampler 











Scores Given 


by 


Trainees* 












1 5 


1 4 1 


3 


t 2 


- 

1 1 1 


0 




6 1 1 


14 


1 2 














1 1 1 










1 3 I 


13 1 




Intended 


5 i : 


1 


1 10 


1 5 1 










Scores 


2 1 ! 








4 


1 12 I 








4 1 1 




1 1 


i 10 1 


4 


1 1 1 








3 1 1 


1 


1 1 




14 










0 1 1 












1 1 


15 



*The total niuaber of kindergarten trainees Included one central office adminis- 
trator who did not participate as a scorer. 
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Appendix G 





Tally of Scores 


on First 


-Grade Training SamDles 








Scores Given bv Trainees* 






II 6 1 s 


1 4 


1 3 


12 111 


0 




S M 18 i 4 


1 2 










111 1 




1 1 


6 1 17 1 




x ax kcuucu 


Oil 7 I 14 


1 2 


_J _1 1 






Scores 


2 11 ! 


1 1 


1 2 1 


16 1 5 ! 






4 II IB 


1 15 


1 4 I 








3 M 11 


1 3 


I 17 1 


112 1 






Oil 1 








24 



•The total nuiaber of first-grade trainees included one central office adminis 
trator Mho did not participate as a scorer. 
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